Task circumstance processing device and method

ABSTRACT

A task circumstance processing system includes a processor that executes a process. The process includes: referencing a recognition information stored in a memory, the recognition information stores, for each of plural task processes in task definitions defining relationships between the plural task processes, recognition information for recognizing execution of each of the plural task processes, and extracting for each of the task processes a timing where the recognition information is expressed in observation data from observing circumstances of the task; and outputting a result of comparing a relationship between plural task processes that have been executed as identified by the extracted timings, against a relationship between plural task processes defined by the task definitions stored in the memory.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of theprior Japanese Patent Application No. 2016-068526, filed on Mar. 30,2016, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein are related to a task circumstanceprocessing system, a task circumstance processing method, and a storagemedium storing a task circumstance processing program.

BACKGROUND

Hitherto, video recording technology capable of surveying businessprocesses has been proposed. In such technology, a region of concern isdefined within a field of vision of plural respective cameras that eachhave a defined field of vision, a background image is taken for theregion of concern, and movement in the region of concern is detected bycomparing each frame against the background image. The video recordingis then segmented, and indexes are created based on the movementdetection.

RELATED PATENT DOCUMENTS

-   Japanese Laid-Open Patent Publication No. 2008-47110

SUMMARY

According to an aspect of the embodiments, a task circumstanceprocessing system includes a memory, and a processor coupled to thememory. The processor is configured to: reference a recognitioninformation stored in the memory, the recognition information stores,for each of plural task processes in task definitions definingrelationships between the plural task processes, recognition informationfor recognizing execution of each of the plural task processes, andextract for each of the task processes a timing where the recognitioninformation is expressed in observation data from observingcircumstances of the task; and output a result of comparing arelationship between plural task processes that have been executed asidentified by the extracted timings, against a relationship betweenplural task processes defined by the task definitions stored in a taskdefinition storage section.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot restrictive of the invention.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a functional block diagram illustrating a schematicconfiguration a task circumstance processing device according to a firstto a third exemplary embodiment;

FIG. 2 is a diagram illustrating an example of system data of the firstexemplary embodiment;

FIG. 3 is a diagram illustrating an example of a task definitiondatabase (DB) of the first exemplary embodiment;

FIG. 4 is a diagram illustrating an example of a recognition informationDB of the first exemplary embodiment;

FIG. 5 is a diagram illustrating an example of an index list of thefirst exemplary embodiment;

FIG. 6 is a diagram illustrating an example of an output screen of thefirst exemplary embodiment;

FIG. 7 is a block diagram illustrating a schematic configuration of acomputer that functions as a task circumstance processing deviceaccording to the first to the third exemplary embodiments;

FIG. 8 is a flowchart flow illustrating an example of a procedure duringtask execution by an inspector of the first exemplary embodiment;

FIG. 9 is a flowchart illustrating an example of task circumstanceprocessing of the first to the third exemplary embodiments;

FIG. 10 is a flowchart illustrating an example of index creationprocessing of the first exemplary embodiment;

FIG. 11 is a flowchart illustrating an example of comparison processingof the first exemplary embodiment;

FIG. 12 is a flowchart illustrating an example of comparison resultoutput processing of the first exemplary embodiment;

FIG. 13 is a schematic diagram illustrating an example of a comparisonbetween circumstances of a task and task definitions in the firstexemplary embodiment;

FIG. 14 is a diagram illustrating an example of system data of thesecond exemplary embodiment;

FIG. 15 is a diagram illustrating an example of a task definition DB ofthe second exemplary embodiment;

FIG. 16 is a diagram illustrating an example of a recognitioninformation DB of the second exemplary embodiment;

FIG. 17 is a diagram illustrating an example of index lists of thesecond exemplary embodiment;

FIG. 18 is a diagram illustrating an example of an output screen of thesecond exemplary embodiment;

FIG. 19 is a flowchart flow illustrating an example of a procedureduring task execution by a user of the second exemplary embodiment;

FIG. 20 is a flowchart illustrating an example of index creationprocessing of the second exemplary embodiment;

FIG. 21 is a flowchart illustrating an example of comparison processingof the second exemplary embodiment;

FIG. 22 is a flowchart illustrating an example of comparison resultoutput processing of the second exemplary embodiment;

FIG. 23 is a schematic diagram illustrating an example of a comparisonbetween circumstances of a task and task definitions of the secondexemplary embodiment;

FIG. 24 is a diagram illustrating an example of system data of the thirdexemplary embodiment;

FIG. 25 is a diagram illustrating an example of a task definition DB ofthe third exemplary embodiment;

FIG. 26 is a diagram illustrating an example of a recognitioninformation DB of the third exemplary embodiment;

FIG. 27 is a diagram illustrating an example of index lists of the thirdexemplary embodiment;

FIG. 28 is a diagram illustrating an example of a comparison result listof the third exemplary embodiment;

FIG. 29 is a diagram illustrating an example of an output screen of thethird exemplary embodiment;

FIG. 30 is a flowchart flow illustrating an example of a procedureduring task execution by an operator of the third exemplary embodiment;

FIG. 31 is a flowchart illustrating an example of index creationprocessing of the third exemplary embodiment;

FIG. 32 is a flowchart illustrating an example of comparison processingof the third exemplary embodiment;

FIG. 33 is a flowchart illustrating an example of comparison resultoutput processing of the third exemplary embodiment;

FIG. 34 is a schematic diagram illustrating an example of a comparisonbetween circumstances of a task and task definitions of the thirdexemplary embodiment; and

FIG. 35 is a functional block diagram illustrating a schematicconfiguration of a task circumstance processing system.

DESCRIPTION OF EMBODIMENTS

Detailed explanation follows regarding examples of exemplary embodimentsaccording to technology disclosed herein, with reference to thedrawings.

First Exemplary Embodiment

In the first exemplary embodiment, explanation follows regarding anexample of a case in which circumstances of a task related to inspectionof an installation are analyzed.

As illustrated in FIG. 1, a task circumstance processing device 100according to the first exemplary embodiment compares task circumstancesrecognized from system data 121 and image data 122 against taskdefinitions defined as proper forms of the task, and outputs acomparison result list 129.

The system data 121 is data that records and manages data in a tasksystem employed when executing task processes. The system data 121includes, for example, operation recordings input by an operator and logdata recorded by the task system when the system runs. In the firstexemplary embodiment, an inspector who performs tasks related toinspection of an installation registers an inspection recording in aninspection management system that is a task system. This registeredinformation is input to a task circumstance processing device 100 as thesystem data 121. FIG. 2 illustrates an example of the system data 121 inthe first exemplary embodiment. In the example of FIG. 2, the date onwhich the inspection was performed, and the inspector ID that isidentification information for the inspector who performed theinspection, are associated with each other and registered as aninspection record.

The image data 122 is data, captured by a video camera or the like,imaging a state in which task being executed. The image data 122 is anexample of observation data of technology disclosed herein. In the firstexemplary embodiment, the image data 122 captured from a video cameraworn on a specific location of the inspector (for example, the head)during a task related to inspection of the installation is input to thetask circumstance processing device 100. The image data 122 includesplural frames arranged in a time series in order of capture, and timeinformation is associated with each frame. The time information may bean actual date and time of capture, or may represent time elapsed whencaptured since a reference point of the image data 122 (for example, arecording start time). Here, explanation follows regarding an example ofthe former case.

The task circumstance processing device 100 functionally includes anextraction section 112 and an output section 114. A task definitiondatabase (DB) 125 and a recognition information DB 126 are stored in aspecific storage region of the task circumstance processing device 100.

For each task, the task definition DB 125 stores corresponding taskdefinitions defined by relationships between plural task processesincluded in the task. The relationship between task processes is, forexample, an execution sequence for the respective task processes, acontinuation time of an observed, identified state relating specifictask processes to each other, or a time interval spanning from the endof a given task process until the start of the next task process. Thetask definition describes conditions to be met by one or pluraldefinition items. The condition to be met by the definition item is aform that the task is to take, and is a correct procedure or a numberserving as a target, expressed using a relationship between taskprocesses such as those described above. Moreover, a calculation methodfor calculating values needed for determination of the conditions isexpressed by the definition item using a relationship between taskprocesses such as those described above.

The first exemplary embodiment gives an example in which a task relatedto inspection of installations includes the start of inspection,inspection of installation A, inspection of installation B, inspectionof installation C, and the end of inspection, as the task processes. Thetask definitions of this task, for example, state that installation A,installation B, and installation C are inspected in this order, and thatthe overall time for the inspection is within 10 minutes. An example ofthe task definition DB 125 in this case is illustrated in FIG. 3. In theexample of FIG. 3, the task definitions include definition items No. 1to No. 4. No. 1 to No. 3 are definition items that stipulate aninspection order for the installations, and No. 4 is a definition itemthat stipulates an overall inspection time. Moreover, the “Time (X)” istime information regarding the frame in the image data 122 appended withan index (described in detail later) of item X.

Moreover, in the example of FIG. 3, a “corresponding item” employed whenidentifying the index from the definition item during output of acomparison result, described later, is included in the task definitionDB 125.

An image pattern for recognizing from the image data 122 the place wherethe task process has been executed are stored in the recognitioninformation DB 126 for each task process. The image pattern is, forexample, a pattern expressing a predetermined QR code (registeredtrademark), text string, numeral, or the like, and a different imagepattern is defined for each task process. Moreover, the image pattern isimaged by a video camera worn by the inspector when the inspector startseach task process. FIG. 4 illustrates an example of the recognitioninformation DB 126 in the first exemplary embodiment. The “recognitionresult” of the task process is information indicating which imagepattern was recognized in the image data 122, namely, which task processwas executed. The “content” of the task process is an item convenientlydescribing the content of the task process to facilitate understandingof the explanation, and is not a required item in the recognitioninformation DB 126.

The extraction section 112 extracts each image pattern stored in therecognition information DB 126 from each frame of the input image data122 using image processing such as pattern matching. The extractionsection 112 creates an index that associates time information regardingframes extracted by any image pattern, with the recognition resultcorresponding to the extracted image patterns. FIG. 5 illustrates anexample of an index list 127 storing a time series of created indexes.In the example of FIG. 5, the “date” and “time” are time information ofthe frames, and the “item” is a recognition result corresponding to theimage pattern extracted from the frame identified by the timeinformation indicated by the “date” and “time”. For example, the indexof the second row of the index list 127 of FIG. 5 indicates that imagepattern a was extracted from the frame identified by the date “3/1” andthe time “9:01”. Namely, it can be identified from the index list 127that, in an inspection task of 3/1, the inspector arrived atinstallation A, an image pattern a affixed to the installation A wascaptured, and inspection of the installation A started at 9:01.

Based on the indexes created by the extraction section 112, the outputsection 114 identifies relationships between plural task processesincluded in the inspection task executed at a designated date, andcompares the relationships against relationships between task processesdefined in the task definition DB 125. More specifically, the outputsection 114 acquires, from the index list 127, time information employedby the calculation method and condition of each definition item definedin the task definition DB 125. In cases in which a calculation method isstipulated by the definition item, the output section 114 employs timeinformation acquired from the index list 127, and calculates values forthat definition item according to the stipulated calculation method. Incases in which there is time information acquired from the index list127 and a calculated value, the output section 114 then uses thosevalues to determine whether or not the task circumstances meet thecondition of each definition item. For example, the output section 114appends a determination result “OK” to definition items meeting theconditions, and appends a determination result “BAD” to definition itemsnot meeting the conditions.

For example, the date 3/1 is designated as the inspection task to be thetarget of comparison against task definitions. Moreover, as illustratedin FIG. 3, the condition stipulated by the definition item of No. 1 ofthe task definition DB 125 is “(Time (A)<Time (B)) AND (Time (A)<Time(C))”. The output section 114 accordingly acquires, as the Time (A), thetime “9:01” of the index for which the date is “3/1” and the item is “A”from the index list 127. Similarly, the output section 114 acquires Time(B)=9:11 and Time (C)=9:05. In such cases, the output section 114associates the determination result “OK” with the definition item of No.1 since the condition of the definition item of No. 1 is met.

In the case of the definition item of No. 4 of the task definition DB125 illustrated in FIG. 3, the calculation method is “overall inspectiontime T=Time (end)−Time (start)”. The output section 114 acquires Time(start)=9:00 and Time (end)=9:15 from the index list 127, and calculatesthat the overall inspection time T is 15 minutes. Since the condition ofthe definition item of No. 4 is “T<10 minutes”, the output section 114associates the determination result “BAD” with the definition item ofNo. 4.

The output section 114 acquires an inspector ID corresponding to thedesignated date from the system data 121, and stores, in the comparisonresult list 129, the designated date, the acquired inspector ID, and theassociated comparison result of the comparison against the determinationresult of each definition item. Note that in addition to thedetermination result of each definition item, the output section 114 mayinclude a value of a circumstance for each definition item.

Moreover, the output section 114 displays an output screen that includesthe stored comparison result list 129 on a display device. FIG. 6illustrates an example of an output screen 130. The comparison resultlist 129 and an image playback region 131 are included in the outputscreen 130. Note that although FIG. 6 is an example of the output screen130, the data structure of the comparison result list 129 is asillustrated in FIG. 6. Each row of the comparison result list 129corresponds to a comparison result corresponding to one inspection task.

When the determination result of any definition item included in any ofthe comparison results is selected from the comparison result list 129,the output section 114 identifies the item of the index corresponding tothat definition item from the definition item corresponding to theselected determination results. The item of the index corresponding tothe definition item can be identified by the “corresponding item” of thetask definition DB 125. The output section 114 also identifies the dateof the comparison result included in the selected determination result.The output section 114 then identifies the index that includes the itemand date of the identified index from the index list 127. The outputsection 114 then plays back the corresponding image data 122 in theimage playback region 131 from the place (frame) indicated by the timeinformation of the identified index.

For example, as illustrated in FIG. 6, the determination result “BAD” ofthe definition item of No. 2 in the inspection task of 3/3 (the dashedportion in FIG. 6) has been selected. In this case, the image data 122is played back from a scene capturing the task process for inspectinginstallation B, in the inspection task of 3/3. Note that in cases inwhich the determination result of the definition item of No. 4 has beenselected, since the “corresponding item” for the definition item of No.4 is “start”, the image data 122 is played back from a place (frame)indicated by the time information of the index having the item “start”.

The task circumstance processing device 100 may, for example, beimplemented by the computer 40 illustrated in FIG. 7. The computer 40includes a CPU 41, memory 42 serving as a non-transient storage section,and a non-volatile storage region 43. The computer 40 further includesinput/output devices 44 such as an input device and a display device, aread/write (R/W) section 45 that controls reading and writing of datafrom and to a recording medium 49, and a communication interface (I/F)46. The CPU 41, the memory 42, the storage section 43, the input/outputdevices 44, the R/W section 45, and the communication I/F 46 areconnected to one another through a bus 47.

The storage section 43 may be implemented by a hard disk drive (HDD), asolid state drive (SSD), flash memory, or the like. A task circumstanceprocessing program 150 for causing the computer 40 to function as thetask circumstance processing device 100 is stored in the storage section43, which serves as a storage medium. The task circumstance processingprogram 150 includes an extraction process 152 and an output process154. The storage section 43 further includes an information storageregion 60 that stores information respectively forming the taskdefinition DB 125, the recognition information DB 126, and the indexlist 127.

The CPU 41 reads the task circumstance processing program 150 from thestorage section 43, expands the task circumstance processing program 150into the memory 42, and sequentially executes the processes included inthe task circumstance processing program 150. The CPU 41 operates as theextraction section 112 illustrated in FIG. 1 by executing the extractionprocess 152. The CPU 41 also operates as the output section 114illustrated in FIG. 1 by executing the output process 154. The computer40, which executes the task circumstance processing program 150, therebyfunctions as the task circumstance processing device 100.

Note that the functionality implemented by the task circumstanceprocessing program 150 may be implemented by, for example, asemiconductor integrated circuit, and more specifically, by anapplication identified integrated circuit (ASIC) or the like.

Next, explanation follows regarding operation of the task circumstanceprocessing device 100 according to the first exemplary embodiment.First, prior to the task circumstance processing being executed by thetask circumstance processing device 100, the state during actual taskexecution is observed. In the first exemplary embodiment, for example,due to the inspector executing the inspection, the image data 122 isobtained as the observation data by the procedure expressed by the flowof task execution by the inspector illustrated in FIG. 8. Detailedexplanation follows regarding the flow of task execution by theinspector.

At step S111, the inspector wears the video camera on a specificlocation (for example, the head) of their own body such that the stateof the inspection operation by the inspector is included in the imagecapture range, and starts image capture. Image capture by the videocamera continues until the inspection task ends.

Next, at step S112, the inspector moves to a location where an imagepattern indicating the start of the inspection task is affixed, andimages that starting image pattern.

Next, at step S113, the inspector moves to an installation targeted forinspection, and images an image pattern affixed to that installation foridentifying the installation when having arrived at the installationlocation of that installation. Then, at step S114, the inspectorinspects the installation whose image pattern was imaged at step S113above.

Next, at step S115, the inspector determines whether or not inspectionof all of the installations targeted for inspection has completed, andmoves to the next installation to perform inspection in cases in whichan uninspected installation is present. Namely, the procedure of thestep S113 and step S114 is repeated. When inspection is complete for allof the installations, at the next step S116, the inspector moves to thelocation where the image pattern indicating the end of inspection taskis affixed, images the end image pattern, and stops image capture by thevideo camera.

Next, at step S117, the inspector registers the date on which theinspection was performed, and his own inspector ID, in the inspectionmanagement system as an inspection record. Moreover, at step S118, theimage data 122 captured by the video camera is registered in a librarythat is a specific storage region of the inspection management system oranother device, and the inspection task ends.

Thus, the system data 121 accumulates in the inspection managementsystem and the image data 122 accumulates in the library, due to theinspector executing the inspection task. Then, in the task circumstanceprocessing device 100, when analysis of the task circumstance isinstructed, the task circumstance processing illustrated in FIG. 9 isexecuted in the task circumstance processing device 100.

At step S20, the extraction section 112 executes index creationprocessing, described in detail later, to append indexes correspondingto the execution of task processes to the image data 122. Next, at stepS30, the output section 114 compares the task circumstances against thetask definitions by executing comparison processing, described in detaillater. Next, at step S40, the output section 114 displays comparisonresults on the display device by executing comparison result outputprocessing, described in detail later. Detailed description of eachprocessing is given below.

First, explanation follows regarding the index creation processing, withreference to FIG. 10.

At step S121, the extraction section 112 reads one item of the imagedata 122 registered in the specific library. Next, at step S122, theextraction section 112 selects one frame in order from the leading frameof the read image data 122.

Next, at step S123, the extraction section 112 determines whether or notan image pattern matching any image pattern stored in the recognitioninformation DB 126 is included in the selected frame by image processingsuch as pattern matching. Processing transitions to step S124 in casesin which any image pattern is included. Processing transitions to stepS125 in cases in which no image pattern is included.

At step S124, the extraction section 112 creates an index associatingthe time information regarding the frame selected at step S122 abovewith an item indicating a recognition result, found at step S123,corresponding to the image pattern determined as included in that frame.The extraction section 112 stores the created index in the index list127.

Next, at step S125, the extraction section 112 determines whether or notthe frame selected at step S122 is the final frame of the image data 122read at step S121. Processing transitions to step S126 in cases in whichthe frame is the final frame, and in cases in which the frame is not thefinal frame, processing returns to step S122, and the extraction section112 selects the next frame.

At step S126, the extraction section 112 determines whether or not theprocessing to create the indexes has completed for all of the image data122 registered in the library. Processing returns to step S121 in casesin which unprocessed image data 122 is present, or in cases in which theprocessing has completed for all of the image data 122, the indexcreation processing ends and processing returns to the task circumstanceprocessing (FIG. 9).

Next, explanation follows regarding the comparison processing, withreference to FIG. 11.

At step S131, the output section 114 receives designation of a daterange for the performance date of the target inspection task in whichthe circumstances are compared to task definitions. Next, at step S132,the output section 114 selects one date from the designated date range.

Next, at step S133, the output section 114 acquires the inspector IDcorresponding to the date selected at step S132 above from the systemdata 121.

Next, at step S134, the output section 114 acquires, from the index list127, the indexes that include the time information to be employed tocalculate a value in accordance with the calculation method stipulatedfor each definition item of the task definition DB 125, and determinethe condition.

Next, at step S135, the output section 114 employs the time informationof the acquired index to perform calculation of values in accordancewith the calculation method stipulated for each definition item, anddetermination of the condition, and associates a determination resultwith each definition item.

Next, at step S136, the output section 114 stores, in the comparisonresult list 129, the date selected at step S132 above, the inspector IDacquired at step S133 above, and a comparison result associated with thedetermination result for each definition item obtained at step S135above.

Next, at step S137, the output section 114 determines whether or not allof dates in the date range received at step S131 above have beenselected at step S132 above. Processing returns to step S132 in cases inwhich dates that have not yet been selected are present, or in cases inwhich all of the dates have been selected, the comparison processingends and processing returns to the task circumstance processing (FIG.9).

Next, explanation follows regarding comparison result output processingwith reference to FIG. 12.

At step S141, the output section 114 displays the output screenincluding the stored comparison result list 129 on the display device.

Next, at step S142, the output section 114 receives a selection of adetermination result of any definition item included in any comparisonresult.

Next, at step S143, based on the “corresponding item” of the taskdefinition DB 125, the output section 114 identifies, from thedefinition item corresponding to the received determination result, theitem of the index corresponding to that definition item. Moreover, theoutput section 114 identifies the date of the comparison result thatincludes the selected determination result.

Next, at step S144, the output section 114 identifies the index thatincludes the index item and the date identified at step S143 above, fromthe index list 127.

Next, at step S145, the output section 114 plays back the correspondingimage data 122 on the image playback region 131, from the place (frame)indicated by the time information of the index identified at step S144above. Then, the comparison result output processing ends and the taskcircumstance processing (FIG. 9) also ends.

As explained above, the task circumstance processing device 100according to the first exemplary embodiment appends indexes to imagedata according to task definitions, uses the indexes appended to theimage data to compare the task circumstances to the task definitions,and outputs the comparison results. Analysis of the task circumstancesin comparison with the task definitions can thereby be assisted.

More specifically, FIG. 13 illustrates a schematic example of acomparison of the task circumstance against the task definition ininspection task serving as an example of the first exemplary embodiment.For example, that inspection in the order of installation A,installation B, and installation C is defined in the task definitionssince the action timing does not match, giving rise to a waiting time,unless the inspection of installation B is after the inspection of theinstallation A. In the task definitions that follow, in the comparisonresult list 129 of the output screen 130 illustrated in FIG. 6, theoperation times of the inspection task of 3/3 and the inspection task of3/4 do not meet the condition defined by the task definitions. However,it is not determinable from the operation times alone whether the thereis a problem with the inspection order or a problem with the method ofthe inspection operation at each installation.

In the first exemplary embodiment, displaying the determination resultsassociated with the task definitions like on the output screen 130illustrated in FIG. 6 enables selection of the determination result forwhich confirmation of details is desired, and enables the image data 122of the corresponding place to be confirmed. For example, selecting thedetermination result “BAD” for the definition item for the inspectionorder of the installation B (the dashed portion in FIG. 6) andconfirming the image data enables confirmation to be made that wastefulwaiting time is occurring when not meeting the action timing of theinstallation B during inspection of the installation B. From suchcontents, as illustrated in FIG. 13, for example, since the distancebetween the installation locations of the installation A and theinstallation C is short, the inspection will be performed in order ofthe installation A, the installation C, and the installation B in somecases, and this enables task circumstances that suggest a case in whichinspection time will be lengthened to be ascertained.

Second Exemplary Embodiment

In the second exemplary embodiment, explanation follows regarding anexample of a case in which the circumstances of a task in whichcomputer-aided design (CAD) is employed are analyzed. Note that in thesecond exemplary embodiment, configuration similar to configuration ofthe first exemplary embodiment is allocated reference numerals thatshare the same final two digits, and explanation focuses on portionsthat differ from the first exemplary embodiment.

As illustrated in FIG. 1, a task circumstance processing device 200according to the second exemplary embodiment compares task circumstancesrecognized from system data 221 and image data 222 against taskdefinitions, and outputs a comparison result list 229.

The system data 221 of the second exemplary embodiment is log data,recorded by the task system, of operations in an information processingterminal installed with CAD. FIG. 14 illustrates an example of systemdata 221 in the second exemplary embodiment. In the example of FIG. 14,each row (each record) is an operation log item for one operation. Eachoperation log item includes the time at which a log-on or log-offoperation was performed, information indicating whether the performedoperation was a log-on or a log-off, a user ID that is identificationinformation for a CAD user, and information regarding a terminal ID orthe like that is identification information of the operated informationprocessing terminal.

The image data 222 in the second exemplary embodiment is image datacaptured by a video camera installed to a fixed position capable ofrecognizing the user sitting down at or getting up from the informationprocessing terminal. For example, the video camera is installed on, forexample, a display upper portion of the information processing terminalso as to image the user performing operations while facing the display.

The task circumstance processing device 200 functionally includes anextraction section 212 and an output section 214. Moreover, a taskdefinition DB 225 and a recognition information DB 226 are stored in aspecific storage region of the task circumstance processing device 200.

The task that employs CAD includes, as task processes, acquiring andreleasing a CAD license, and a user sitting down at and getting up fromthe information processing terminal on which CAD is installed, as anexample in the second exemplary embodiment. The task definitions of thistask, for example, state that the operation ratio of CAD while thelicense is reserved is greater than 70%. The operation ratio is theproportion of time in which CAD is being used and is operation in aperiod spanning from when the CAD license is acquired to the CAD licensebeing released, and is determination criteria when determining, forexample, how many licenses are to be contracted.

In such cases, FIG. 15 illustrates an example of the task definition DB225. In the example of FIG. 15, the task definitions include thedefinition items of No. 1 and 2. No. 1 is the definition itemstipulating calculation of the time for which the CAD license isreserved, and No. 2 is a definition item stipulating an operation timeby CAD. The first term in the calculation method is the total timebetween sitting down and getting up, the second term is the time fromlogging on to first getting up, and the third term is the time fromsitting down for the last time and logging off. The condition “T>0.7×No.1” of the definition item of No. 2 expresses a task definition statingthat the operation ratio of CAD while the license is reserved is greaterthan 70%.

As illustrated in FIG. 16, a terminal image correspondence table 226A,an operation recognition table 226B, and an image pattern recognitiontable 226C are included in the recognition information DB 226 in thesecond exemplary embodiment.

The terminal image correspondence table 226A is a table indicatingcorrespondence relationships indicating whether each image data 222 hascaptured the user sitting down at or getting up from any informationprocessing terminal. The terminal ID of the information processingterminal and the image ID, which is identifying information of the imagedata 222, are associated and registered in the terminal imagecorrespondence table 226A. The image ID may, for example, be informationunique to the video camera that captured each image data 222.

An operation recognition table 226B stores, for each task process,operations for recognizing places where the task process was executedfrom the system data 221. Task processes recognized from the system data221 are acquisition and release of the CAD license. The operationrecognition table 226B of FIG. 16 indicates that acquisition of thelicense is recognized when logged on to the information processingterminal, and release of the license is recognized when logging off.

An image pattern recognition table 226C is similar to the recognitioninformation DB 126 of the first exemplary embodiment, and in the secondexemplary embodiment, patterns are stored for recognizing that the userhas sat down at or gotten up from an information processing terminal byimage processing.

Similarly to the extraction section 112 of the first exemplaryembodiment, the extraction section 212 uses the image patterns stored inthe image pattern recognition table 226C to create indexes. Theextraction section 212 stores the created indexes in an index list 227for each item of image data 222.

The extraction section 212 extracts operation log items having loggingon as the “operation” from the system data 221, and creates indexesassociating a “time” with an item “acquisition” of that operation log.Similarly, the extraction section 212 extracts operation logs havinglogging off as the “operation” from the system data 221, and createsindexes associating a “time” with an item “release” of that operationlog. The extraction section 212 references the terminal imagecorrespondence table 226A and identifies the image data 222corresponding to the “terminal ID” of the operation log, and stores theindexes created from the system data 221 in the index list 227 of theidentified image data 222.

FIG. 17 illustrates an example of index lists 227 pairing the indexescreated from the image data 222 and the indexes created from the systemdata 221. In this manner, indexes of the items “acquisition” and“release” can be simply created using the system data 221 withoutperforming image processing to recognize the image pattern.

The output section 214 extracts operation log item groups of logging onand logging off to and from each information processing terminal fromthe system data 221. Groups of operation log items of logging on andlogging off are each a group of an operation log item of logging on andan operation log, appearing immediately after that operation log item,of logging off from the same information processing terminal by the sameuser. Moreover, the output section 214 identifies the time correspondingto logging on (referred to as the “log-on time” hereafter) and the timecorresponding to logging off (referred to as the “log-off time”hereafter) included in the extracted group of operation log items. Then,the output section 214 references the terminal image correspondencetable 226A and identifies the image data 222 corresponding to the“terminal ID” included in the extracted group of operation log items.The output section 214 extracts, from the index list 227 of theidentified image data 222, indexes for which time information includedin the index is included in a period spanning from the log-on time tothe log-off time.

Similarly to the output section 114 of the first exemplary embodiment,the output section 214 uses the time information included in theextracted indexes to determine the calculation and condition of eachdefinition item defined by the task definition DB 225. Moreover, theoutput section 214 extracts, from the system data 221, the user ID,operation, and time corresponding to each information processingterminal. Then, the output section 214 stores, in the comparison resultlist 229, a comparison result associating the terminal ID, the user ID,the log-in time and log-off time of each user, and the determinationresult of each definition item with one another. Note that thedefinition item associated with the determination result is a definitionitem stipulating a “condition”, and no determination result isassociated with the definition item of No. 1, which stipulates a“calculation method” alone. Accordingly, in the second exemplaryembodiment, the determination result is associated with the definitionitem of No. 2 alone.

Moreover, the output section 214 displays an output screen including thestored comparison result list 229 on the display device. FIG. 18illustrates an example of an output screen 230. The comparison resultlist 229, an outline display region 232, and a usage state listingdisplay region 233 are included in the output screen 230.

When any comparison result (for example, the dashed portion of FIG. 18)is selected from the comparison result list 229, the output section 214identifies a terminal ID, user ID, log-on time, and log-off timeincluded in the selected comparison result. The output section 214 thendisplays the identified information in the outline display region 232.The output section 214 also references the terminal image correspondencetable 226A and identifies the image data 222 corresponding to theidentified terminal ID. The output section 214 extracts, from the indexlist 227 of the identified image data 222, indexes for which the timeinformation included in the index is included in a period spanning fromthe identified log-on time to the log-off time. The output section 214then lists and displays the extracted indexes in a usage state listingdisplay region 233. Since indexes include items according to the taskprocesses, the usage state of the information processing terminal can beascertained by just displaying a listing of indexes.

The task circumstance processing device 200 may, for example, beimplemented by the computer 40 illustrated in FIG. 7. A taskcircumstance processing program 250 for causing the computer 40 tofunction as the task circumstance processing device 200 is stored in thestorage section 43 of the computer 40. The task circumstance processingprogram 250 includes an extraction process 252 and an output process254. Moreover, the storage section 43 includes an information storageregion 60 that stores information respectively forming the taskdefinition DB 225, the recognition information DB 226, and the indexlist 227.

The CPU 41 reads the task circumstance processing program 250 from thestorage section 43, expands the task circumstance processing program 250into the memory 42, and sequentially executes the processes included inthe task circumstance processing program 250. The CPU 41 operates as theextraction section 212 illustrated in FIG. 1 by executing the extractionprocess 252. The CPU 41 also operates as the output section 214illustrated in FIG. 1 by executing the output process 254. The computer40, which executes the task circumstance processing program 250, therebyfunctions as the task circumstance processing device 200.

Note that the functionality implemented by the task circumstanceprocessing program 250 may, for example, be implemented by asemiconductor integrated circuit, and more specifically, by an ASIC orthe like.

Next, explanation follows regarding operation of the task circumstanceprocessing device 200 according to the second exemplary embodiment. Inthe second exemplary embodiment, prior to the task circumstanceprocessing device 200 executing the task circumstance processing, theimage data 222 is, for example, obtained as observation data byexecuting a task using the procedure indicating the flow during taskexecution by the user illustrated in FIG. 19. Detailed explanationfollows regarding the flow during task execution by the user. Note thatwhile the user is executing the task, the information processingterminal surroundings are always being captured by the video camerainstalled in a fixed position.

At step S211, the user logs on to the information processing terminal.Then, at the next step S212, the user performs a task such as using CADin a design operation. The operation continues until the user determinesthat the task has ended at step S231. During this period the user getsup from the information processing terminal and then sits down again toresume the operation. When the task has ended, at step S214, the userlogs off from the information processing terminal.

Thus, the operation logs of each information processing terminal areaccumulated as system data 221, and the image data 222 captured by theeach video camera is accumulated. Note that, similarly to in the firstexemplary embodiment, the image data captured by each video camera isstored in a specific library. Then, in the task circumstance processingdevice 200, when analysis of the task circumstances is instructed, thetask circumstance processing illustrated in FIG. 9 is executed in thetask circumstance processing device 200. Detailed description followsregarding index creation processing, comparison processing, andcomparison result output processing executed by in task circumstanceprocessing of the second exemplary embodiment.

First, explanation follows regarding the index creation processing ofthe second exemplary embodiment, with reference to FIG. 20.

At step S221, the extraction section 212 selects one terminal ID,references the terminal image correspondence table 226A to identify theimage data 222 corresponding to the selected terminal ID, and reads theidentified image data 222 from the library.

Next, similarly to in steps S122 to S125 of the index creationprocessing of the first exemplary embodiment, at steps S222 to S225,indexes are created based on the image patterns.

Next, at step S226, the extraction section 212 extracts, from the systemdata 221, operations log items having log-on as the “operation” andcreates indexes associating the “time” of the operation log item withthe item “acquisition”. Similarly, the extraction section 212 extracts,from the system data 221, operation log items having log-off as the“operation” and creates indexes associating the “time” of that operationlog item with the item “release”. The extraction section 212 pairsindexes created at step S224 above with indexes created at step S226above, and creates the index list 227 of image data corresponding to theterminal ID selected at step S221 above.

Next, at step S228, the extraction section 212 determines whether or notall of the terminal IDs have been selected. Processing returns to stepS221 in cases in which an unselected terminal ID is present, or in casesin which all of the terminal IDs have been selected, the index creationprocessing ends and processing returns to the task circumstanceprocessing (FIG. 9).

Next, explanation follows regarding comparison processing, withreference to FIG. 21.

At step S231, the output section 214 selects one terminal ID, andextracts, from the system data 221, operation log items having a“terminal ID” that matches the selected terminal ID.

Next, at step S232, the output section 214 extracts, from the operationlog extracted at step S231 above, one group of operation log items of alog-on and a log-off.

Next, at step S233, the output section 214 references the terminal imagecorrespondence table 226A and identifies the image data 222corresponding to the terminal ID selected at step S231 above. The outputsection 214 then extracts, from the index list 227 of the identifiedimage data 222, indexes for which time information included in the indexis included in a period spanning from the log-on time to the log-offtime of the group of operation log items extracted at step S232 above.

Next, at step S234, the output section 214 determines whether or not theprocessing to extract indexes has completed for the group of operationlog items of a log-on and a log-off included in the operation log itemsextracted at step S231 above. Processing returns to step S232 in casesin which an unprocessed group of operation log items is present, orprocessing transitions to step S235 in cases in which processing hascompleted for all of the groups of operation log items.

At step S235, the output section 214 uses the time information includedin all of the indexes extracted at step S234 above to determine thecalculation and condition of each definition item defined in the taskdefinition DB 225.

Next, at step S236, the output section 214 extracts, from the systemdata 221, the user ID, operation, and time included in the operation logitems that include the terminal ID selected at step S231 above. Theoutput section 214 then stores, in the comparison result list 229, thecomparison result associated with the terminal ID, the user ID, thelog-in time and log-off time of each user, and the determination resultof each definition item.

Next, at step S237, the extraction section 212 determines whether or notall of the terminal IDs have been selected. Processing returns to stepS231 in cases in which an unselected terminal ID is present, or in casesin which all of the terminal IDs have been selected, the comparisonprocessing ends and processing returns to the task circumstanceprocessing (FIG. 9).

Next, explanation follows regarding comparison result output processingwith reference to FIG. 22.

At step S241, the output section 214 displays the output screen thatincludes the stored comparison result list 229 on the display device.Next, at step S242, the output section 214 receives a selection of anycomparison result.

Next, at step S243, the output section 214 identifies the terminal ID,the user ID, the log-on time, and the log-off time included in theselected comparison result.

Next, at step S244, the output section 214 references the terminal imagecorrespondence table 226A and identifies the image data 222corresponding to the identified terminal ID. The output section 214 thenextracts, from the index list 227 of the identified image data 222,indexes for which time information included in the index is included inthe identified period spanning from the log-on time to the log-off time.

Next, at step S245, the output section 214 displays the informationidentified at step S243 above in the outline display region 232. Theoutput section 214 also lists and displays the indexes extracted at stepS244 above in the usage state listing display region 233.

As explained above, the task circumstance processing device 200according to the second exemplary embodiment also uses informationobtained from the system data to append the indexes to the image data.Thus, in addition to the advantageous effects of the first exemplaryembodiment, information useful for simple processing can also be createdas indexes.

More specifically, FIG. 23 schematically illustrates an example of acomparison between a task circumstance and a task definition, in whichCAD is employed as an example in the second exemplary embodiment. Forexample, from the comparison result list 229 of the output screen 230illustrated in FIG. 18, when a “BAD” comparison result (the dashedportion in FIG. 18) of the determination result is selected, a usagestate of the user indicated by that comparison result is displayed inthe usage state listing display region 233. Based on the usage state,for example, investigations of improvement plans, such as revising atask time proportion in which the usage of CAD can be more concentratedcan be performed.

Note that in the second exemplary embodiment, image data in which pluralinformation processing terminals are included in the capture range of avideo camera may be employed. In such cases, a region representing eachinformation processing terminal can be defined on the image indicated bythe image data, and a user sitting down at and getting up from eachinformation processing terminal may be recognized from the region ofeach information processing terminal. Moreover, in such cases, theterminal IDs and identifying information regarding a region defined foreach image may be associated with each other in the terminal imagecorrespondence table 226A.

Third Exemplary Embodiment

In the third exemplary embodiment, explanation follows regarding anexample of a case in which analysis is performed on circumstances of aproduction task, in particular, a production task that needs apreparation operation performed manually by an operator. Note that inthe third exemplary embodiment, configuration similar to configurationof the first exemplary embodiment is allocated reference numerals thatshare the same final two digits, and explanation focuses on portionsthat differ from the first exemplary embodiment.

As illustrated in FIG. 1, a task circumstance processing device 300according to the third exemplary embodiment compares task circumstancesrecognized from system data 321 and image data 322, against taskdefinitions, and outputs a comparison result list 329.

The system data 321 of the third exemplary embodiment is log dataindicating a running state of each production installation recorded bythe task system. FIG. 24 illustrates an example of the system data 321of the third exemplary embodiment. In the example of FIG. 24, each row(each record) is one running state log item. Each running state log itemincludes a time at which the installation started or stopped running, aninstallation ID that is identifying information of the target productioninstallation, information indicating whether the running state isstarted or stopped, and the like.

Similarly to in the first exemplary embodiment, the image data 322 inthe third exemplary embodiment is image data captured while the operatoris performing an operation, by a video camera worn on a specificlocation (for example, the head) of the operator.

The task circumstance processing device 300 functionally includes anextraction section 312 and an output section 314. A task definition DB325 and a recognition information DB 326 are stored in specific storageregion of the task circumstance processing device 300.

The operator arriving and staying at the installation location of eachproduction installation, and starting and stopping of running of eachproduction installation, are included as task processes in theproduction task, in an example of the third exemplary embodiment. Morespecifically, as the flow of the task, the operator mans pluralproduction installations, and when the operator arrives at a givenproduction installation, the operator starts the running of theproduction installation after having performed a preparation operationat that production installation, and moves to another productioninstallation. When a predetermined production process ends at aproduction installation where running has started, the running thenstops. The task definitions of the task, for example, state that apermissible time from stopping the running of the productioninstallation until the operator arrives is within 5 minutes.

In such cases, FIG. 25 illustrates an example of the task definition DB325. In the example of FIG. 25, the task definitions include adefinition item No. 1 that stipulates a calculation method for anoperator waiting time and a condition determining whether or not theoperator waiting time is less than 5 minutes. The operator waiting timeis stipulated as the time from the production installation stopped untilthe operator arrives.

As illustrated in FIG. 26, an operator image correspondence table 326A,an image pattern recognition table 326C, and a recognition conditiontable 326D are included in the recognition information DB 326 of thethird exemplary embodiment.

The operator image correspondence table 326A is a table indicatingcorrespondence relationships indicating which video camera worn on anoperator captured each item of the image data 322. In the example ofFIG. 26, the operator ID, which is identification information of theoperator, and an image ID, which is identification information of theimage data 322, are associated with each other and registered.

The image pattern recognition table 326C is similar to the recognitioninformation DB 126 of the first exemplary embodiment, and stores adifferent image pattern for each production installation. The imagepatterns are captured by the video camera worn by the operator when theoperator has arrived at each production installation. However, the imagepattern may also be inadvertently captured while the operator isperforming a preparation operation at the production installation.Namely, since execution of a task process is not recognizable from theextracted image pattern alone, installation IDs, associated with imagepatterns, of the production installation indicated by the recognizedimage pattern are associated as a recognition result candidate in theimage pattern recognition table 326C.

The recognition condition table 326D is a table determining conditionsfor determining whether the recognition result candidate recognized byeach image pattern is “arrival” or “staying”. More specifically,configuration is made such that the recognition result is determined as“arrival” in cases in which the image pattern extracted immediatelypreviously and the image pattern extracted the current time aredifferent, and the recognition result is determined as “staying” whenthe image pattern extracted immediately previously and the image patternextracted the current time are the same.

Similarly to the extraction section 112 of the first exemplaryembodiment, the extraction section 312 extracts each image patternstored in the image pattern recognition table 326C from each frame ofthe input image data 322 by image processing such as pattern matching.The extraction section 312 then obtains the recognition result candidateindicating the installation ID. The extraction section 312 alsodetermines that the recognition result is “arrival” or “staying”according to the recognition condition table 326D. Namely, therecognition result is established in accordance with the installation IDobtained as the recognition result candidate. For example, in cases inwhich the recognition result candidate is “10” and “arrival” isdetermined in accordance with the recognition condition table 326D, therecognition result is established as “arrival at production installationhaving installation ID=10”. The extraction section 312 creates an indexthat associates the time information of the frame from which any imagepattern was extracted, with the established recognition result. Theextraction section 312 stores the created index in the index list 327for each item of the image data 322. FIG. 27 illustrates an example ofthe index lists 327. In the example of FIG. 27, installation IDs anditems indicating arrival or staying are associated with the timeinformation of the frames.

The output section 314 extracts groups of running state log items ofstopping and starting each production installation from the system data321. The group of running state log items of stopping and starting isgroup of a running state log item of stopping and a running state logitem of starting of the same production installation appearingimmediately after that running state log item. The output section 314also identifies a time corresponding to stopping included in theextracted group of running state log items (referred to as the “stoptime” hereafter) and a time corresponding to the starting (referred toas the “start time” hereafter). The output section 314 then extractsindexes that include the installation ID of the target productioninstallation from the index lists 327 of all of the image data 322. Theoutput section 314 also narrows the indexes down to the indexes forwhich the time information included in the extracted index is includedin a period spanning from the stop time to the start time.

The output section 314 then extracts indexes having arrival as the“item” from out of the narrowed down indexes, the time information ofthese indexes is employed as the Time (arrival) defined in the taskdefinition DB 325. The output section 314 also employs the “time” of therunning state log item of stopping included in the group of runningstate log items of stopping and starting as the Time (stop) defined inthe task definition DB 325. Then, the determination of the calculationand the condition of the definition items defined in the task definitionDB 325 are performed.

The output section 314 identifies whether indexes that include timeinformation employed as the Time (arrival) were extracted from the indexlists 327 of any image data 322. The output section 314 then referencesthe operator image correspondence table 326A and identifies the operatorID corresponding to the identified image data 322.

The output section 314 stores comparison results associated with theinstallation ID, the Time (arrival), the identified operator ID, and thedetermination result of the definition item in, for example, acomparison result list 329 like that illustrated in FIG. 28.

The output section 314 also displays an output screen including theinformation of the stored comparison result list 329 on the displaydevice. FIG. 29 illustrates an example of an output screen 330. Aninstallation designation article 334 capable of designating a productioninstallation using a pull-down menu or the like, an installation statedisplay region 335, an operator state display region 333, and an imageplayback region 331 are included in the output screen 330.

When the production installation is designated by the installationdesignation article 334, the output section 314 extracts, from thesystem data 321, times and running states of running state log itemsthat include the installation ID of the designated productioninstallation. The output section 314 also extracts the time and item ofthe indexes that include the designated installation ID from the indexlists 327 of all of the image data 322. The output section 314 thenarranges the extracted times and running states or items into a timeseries, and displays this in the installation state display region 335.The “state” of the installation state display region 335 illustrated inFIG. 29 is the running state extracted from the system data 321, or theitem of the index. Moreover, in cases in which the “state” is “arrival”,the output section 314 extracts, from the comparison result list 329,the determination result of the operator ID and the definition item, anddisplays the extracted determination result on the installation statedisplay region 335, using the “time” of that row as a key.

When any operator ID (for example, the dashed portion of FIG. 29) isselected from the installation state display region 335, the outputsection 314 references the operator image correspondence table 326A andidentifies the image data 322 corresponding to the selected operator ID.The output section 314 then displays the index list 327 of theidentified image data 322 on the operator state display region 333 inassociation with the selected operator ID. Moreover, the output section314 emphatically displays the index corresponding to the row of theoperator state display region 333 selected by the operator ID in theinstallation state display region 335. Emphatic display is representedby the shading in the operator state display region 333 illustrated inFIG. 29.

Moreover, the output section 314 plays back, on the image playbackregion 331, the image data 322 identified for displaying the operatorstate display region 333 from the place (frame) indicated by the timeinformation included in the index corresponding to the emphaticallydisplayed index. The index corresponding to the emphatically displayedindex may, for example, be the index one row prior to the emphaticallydisplayed index. Since the emphatically displayed index indicates“arrival”, when arrival at a given production installation is late, theoperation state prior to this can be considered the cause of thelateness. Thus, confirmation of operation states that have a highprobability of having problems is simplified by playing back from theframe indicated by the time information of the index that includes thetime information prior to the emphatically displayed index.

The task circumstance processing device 300 may, for example, beimplemented by the computer 40 illustrated in FIG. 7. A taskcircumstance processing program 350 for causing the computer 40 tofunction as the task circumstance processing device 300 is stored in thestorage section 43 of the computer 40. The task circumstance processingprogram 350 includes an extraction process 352 and an output process354. Moreover, the storage section 43 includes an information storageregion 60 that stores information respectively forming the taskdefinition DB 325, the recognition information DB 326, and the indexlist 327.

The CPU 41 reads the task circumstance processing program 350 from thestorage section 43, expands the task circumstance processing program 350into the memory 42, and sequentially executes the processes included inthe task circumstance processing program 350. The CPU 41 operates as theextraction section 312 illustrated in FIG. 1 by executing the extractionprocess 352. The CPU 41 also operates as the output section 314illustrated in FIG. 1 by executing the output process 354. The computer40, which executes the task circumstance processing program 350, therebyfunctions as the task circumstance processing device 300.

Note that the functionality implemented by the task circumstanceprocessing program 350 may, for example, be implemented by asemiconductor integrated circuit, and more specifically, by an ASIC orthe like.

Next, explanation follows regarding operation of the task circumstanceprocessing device 300 according to the third exemplary embodiment. Inthe third exemplary embodiment, prior to the task circumstanceprocessing being executed by the task circumstance processing device300, for example, due to the task executing, the image data 322 isobtained as the observation data by the procedure illustrated in theflow of task execution by the operator illustrated in FIG. 30. Detailedexplanation follows regarding the flow of task execution by theoperator.

At step S311, the operator wears the video camera on a specific location(for example, the head) of their own body such that the state of thepreparation operation by the operator is included in the image capturerange, and starts image capture. Image capture by the video cameracontinues until the production task ends.

Next, at step S312, the operator moves to any of the productioninstallations manned by the operator himself, and when having arrived atthe installation location of the production installation, images animage pattern, affixed to the production installation, for identifyingthat production installation. Then, at step S313, the operator performsthe preparation operation in the production installation whose imagepattern was imaged at step S312 above. Next, at step S314, the operatorstarts running the production installation for which the preparationoperation has finished.

Next, at step S315, the operator determines whether or not theproduction task has completed. In cases in which the production task hasnot completed, at step S316, the operator moves to the next installationand repeats the preparation operation. Namely, the procedure of stepsS312 to S314 is repeated. In cases in which the production task hascompleted, at the next step S317, the image data 322 captured by thevideo camera is registered in a specific library, and the productiontask ends.

Thus, the image data 322 is accumulated in the library by the operatorexecuting the production task. Moreover, the system data 321 indicatingthe running state of each production installation is also accumulated.Then, in the task circumstance processing device 300, when analysis ofthe task circumstance is instructed, the task circumstance processingillustrated in FIG. 9 is executed in the task circumstance processingdevice 300. Detailed description follows regarding index creationprocessing, comparison processing, and comparison result outputprocessing executed by in task circumstance processing of the thirdexemplary embodiment.

First, explanation follows regarding the index creation processing ofthe third exemplary embodiment, with reference to FIG. 31.

At step S321, the extraction section 312 selects one operator ID,references the operator image correspondence table 326A, identifies theimage data 322 corresponding to the selected operator ID, and reads theidentified image data 322 from the library.

Next, at steps S322 and S323, similarly to at steps S222 and S223 of theindex creation processing of the second exemplary embodiment,determination is made as to whether or not any image patterns to bestored in the image pattern recognition table 326C are included in theselected frames. Processing transitions to step S324 in cases in whichan image pattern is included, or processing transitions to step S326 incases in which no image patterns are included.

At step S324, the extraction section 312 acquires the recognition resultcandidate indicating the installation ID corresponding to the extractedimage pattern. The extraction section 312 then determines whether therecognition result is “arrival” or “staying” in accordance with therecognition condition table 326D, and establishes the recognition resultin accordance with the installation ID obtained as the recognitionresult candidate.

Next, at step S325, the extraction section 312 creates an index thatassociates the time information of the frames from which any imagepatterns were extracted, with the established recognition result.

Next, at step S326, similarly to at step S225 of the index creationprocessing of the second exemplary embodiment, the extraction section312 determines whether or not the selected frame is the final frame,processing transitions to step S327 in cases in which the frame is thefinal frame.

At step S327, the extraction section 312 stores the created indexes inthe index list 327 for each image data 322.

Next, at step S328, the extraction section 312 determines whether or notall of the operator IDs have been selected. Processing returns to stepS321 in cases in which an unselected operator ID is present, or in casesin which all of the operator IDs have been selected, the index creationprocessing ends and processing returns to the task circumstanceprocessing (FIG. 9).

Next, explanation follows regarding the comparison processing withreference to FIG. 32.

At step S331, the output section 314 selects one installation ID, andextracts, from the system data 321, running state log items for whichthe “installation ID” matches the installation ID selected.

Next, at step S332, the output section 314 extracts, from the runningstate log items extracted at step S331 above, one group of running statelog items of stopping and starting.

Next, at step S333, the output section 314 identifies the stop time andthe start time included in the group of running state log itemsextracted at step S332 above. The output section 314 then extracts, fromthe index lists 327 of all of the image data 322, indexes that includethe installation ID selected at step S331 above. The output section 314then narrows down the indexes to indexes for which the time informationincluded in the extracted index is included in a period spanning fromthe stop time to the start time identified in the current step.

Next, at step S334, the output section 314 extracts, from amongst theindexes narrowed down at step S333 above, indexes having arrival as the“item”. Next, at step S335, the time information of the indexesextracted at step S334 above, and the “time” of the running state logitem of stopping extracted at step S332 above are employed to performthe determination of the calculation and condition of the definitionitem defined in the task definition DB 325.

Next, at step S336, the output section 314 references the operator imagecorrespondence table 326A and identifies the operator ID correspondingto the index that includes the time information employed as the Time(arrival). The output section 314 then stores, in the comparison resultlist 329, a comparison result associated with the installation IDselected at step S331 above, the Time (arrival), the identified operatorID, and the determination result of the definition item.

Next, at step S337, the output section 314 determines whether or not theprocessing to extract indexes has completed for the group of operationlog items of starting and stopping included in the running state logitems extracted at step S331 above. Processing returns to step S332 incases in which an unprocessed group of running state log items ispresent, or processing transitions to step S338 in cases in whichprocessing has completed for all of the groups of running state logitems.

At step S338, the extraction section 312 determines whether or not allof the installation IDs have been selected. Processing returns to stepS331 in cases in which an unselected installation ID is present, or incases in which all of the installation IDs have been selected, thecomparison processing ends and processing returns to the taskcircumstance processing (FIG. 9).

Next, explanation follows regarding the comparison result outputprocessing, with reference to FIG. 33.

At step S341, the output section 314 displays the installationdesignation article 334 of the output screen 330, and receivesdesignation of the installation ID.

Next, at step S342, the output section 314 extracts, from the systemdata 321, the time and running state of the running state log items thatinclude the designated installation ID. The output section 314 alsoextracts, from the index lists 327 of all of the image data 322, thetime and item of the indexes that include the designated installationID.

Next, at step S343, the output section 314 orders the times and runningstates or items extracted at step S342 above in a time series, anddisplays the ordered times and running states or items in theinstallation state display region 335. In cases in which the “state” is“arrival”, the output section 314 also extracts the determination resultof the operator ID and the definition item from the comparison resultlist 329 using the time of that row as a key, and displays extracteddetermination result in the installation state display region 335.

Next, at step S344, the output section 314 receives any operator ID (forexample, the dashed portion of FIG. 29) selected from the installationstate display region 335. Next, at step S345, the output section 314references the operator image correspondence table 326A and identifiesthe image data 322 corresponding to the selected operator ID. The outputsection 314 then displays the index list 327 of the identified imagedata 322 on the operator state display region 333 in association withthe selected operator ID. Moreover, the output section 314 emphaticallydisplays the index corresponding to the row of the operator statedisplay region 333 selected by the operator ID in the installation statedisplay region 335.

Next, at step S346, the output section 314 plays back, on the imageplayback region 331, the image data 322 identified for displaying theoperator state display region 333 from the place (frame) indicated bythe time information included in the index corresponding to theemphatically displayed index.

As explained above, the task circumstance processing device 300according to the third exemplary embodiment, can determine the conditiondefined by the task definitions in accordance with the group of theinformation obtained from the system data and the index of the imagedata. Thus, in addition to the advantageous effects of the firstexemplary embodiment, various conditions can be defined in the taskdefinitions.

More specifically, FIG. 34 schematically illustrates an example of acomparison between the circumstance of production task and taskdefinitions, as an example of the third exemplary embodiment. Forexample, the running state of the installation and the operation stateof the operator can be confirmed by the output screen 330, like thatillustrated in FIG. 29. Thus, whether there is a problem with theoperation method itself, whether there is a problem with workerdistribution or operation distribution, such as the number of productioninstallations manned by the operator, and the like can be ascertained,and a review of task planning can be assisted.

In each of the exemplary embodiments above, indexes indicating that taskprocesses were executed are created from observation data such as imagedata, and task analysis is also simplified when modifying taskdefinitions in a method for using the indexes to determine the conditiondefined by the task definitions.

More specifically, new definition items are added to the taskdefinition, and recognition information, such as an image pattern forcreating the indexes employed in the calculation method or conditionstipulated by the definition item, is newly added to the recognitioninformation DB. In such cases, indexes related to newly added imagepatterns may be created from image data stored in the library, withoutre-capturing the task circumstance with the video camera. For example,in the first exemplary embodiment, at step S123 of the index creationprocessing, in cases in which whether or not the added image pattern isincluded in each frame is determined alone and the image pattern isincluded, a new index is created and the new index may be added to analready created index list. Then, the comparison processing illustratedin FIG. 11 may be executed based on the new task definitions thatinclude the added definition item.

Moreover, in cases in which a new definition item that employed an imagepattern already stored in the recognition information DB has been addedto the task definitions, the already created index list may be employed,and the comparison processing illustrated in FIG. 11 may be executedbased on the new task definitions that include the added definitionitem.

Although explanation has been given regarding cases in which the imagedata is employed as the observation data in each of the exemplaryembodiments above, there is no limitation thereto. For example, voicedata that is a collection of sounds from the operation site or audiodata that is a recording of a voice during a telephone conversation maybe employed as the observation data. In such cases, speaking apredetermined word such as “OK”, or a predetermined sound such as abuzzer, may be stored in the recognition information DB as the patternindicating that the task has been executed.

Moreover, a sensor value observed by a thermometer installed in at, forexample, the operation site, or a sensor such as a hygrometer may beemployed as the observation data. In such cases, a predeterminedthreshold value or time series changes or the like in the sensor valuemay be stored in the recognition information DB as a pattern indicatingthat the task has been executed. Moreover, instead of an image patternaffixed to the installation like in the explanation of each exemplaryembodiment above, execution of task processes may be recognized based oninformation emitted from an emitter such as a beacon.

Moreover, task processes recognized from observation data is not limitedto the examples of each exemplary embodiment above. For example,configuration may be made to recognize an action of extending the handto a specific position (placing an item), an action such as pointing andconfirming, passing a target object, increase or decrease of a targetobject, a change in a display object such as a PATLITE (registeredtrademark), or the like.

In each exemplary embodiment above, explanation has been given regardingcases in which the result of comparing task circumstance and the taskdefinitions is displayed on a display device; however, output resultsare not limited to being displayed. For example, printing output,storage to a storage medium, or output to an external device may beperformed.

Moreover, although explanation has been given in each exemplaryembodiment above regarding cases in which the extraction section and theoutput section are implemented by one computer, there is no limitationthereto. For example, as illustrated in FIG. 35, configuration may bemade using a task circumstance processing system 400 including pluralbase devices that each include an extraction section 412 and amanagement device 402 that includes an output section 414. In suchcases, observation data such as image data may be stored in a localstorage section connected to each base device 404, and an index listcreated by the extraction section 412 may be transmitted to themanagement device 402. Network loads can be reduced between themanagement device 402 and each base device 404 by transmitting only aneeded portion of image data having a large data size and the like.Moreover, even in cases in which the communication cost is high, such asin cases in which the base device is disposed overseas, the data amounttransmitted to the management device 402 is suppressed, reducing thecommunication cost.

As in each exemplary embodiment above, in the comparison with the taskdefinitions, circumstances of task actually occurring can be ascertainedby analyzing the task circumstances, and deviation from the taskdefinitions can be prevented. Moreover, understanding and acceptance bysite operators toward proposals for improvement plans and the like isfacilitated since task circumstances can be accurately ascertained.Moreover, in cases in which there are problems with the circumstances ofthe task, autonomous improvements to the task can be more easilyproposed at the operation site since the operation states related to thecause can be ascertained. Moreover, task circumstances in whichdifferences from the task definitions arise can be confirmed, and casesin which a method in which the task circumstances are more favorablethan in the task definitions or the like can be ascertained, and thisenables a review of the task definitions to be assisted.

Although explanation has been given above regarding a mode in which thetask circumstance processing program 150, 250, 350 is pre-stored(installed) in the storage section 43, the task circumstance processingprogram 150, 250, 350 may also be provided in a mode recorded to astorage medium such as a CD-ROM or DVD-ROM.

The related technology merely analyzes circumstances of a businessprocess, such as whether or not an employee is in their assigned regionand the number of customers in each location, based on the segmentedvideo recording.

According to technology disclosed herein, analysis of task circumstancesin comparison with task definitions can be assisted.

All examples and conditional language provided herein are intended forthe pedagogical purposes of aiding the reader in understanding theinvention and the concepts contributed by the inventor to further theart, and are not to be construed as limitations to such specificallyrecited examples and conditions, nor does the organization of suchexamples in the specification relate to a showing of the superiority andinferiority of the invention. Although one or more embodiments of thepresent invention have been described in detail, it should be understoodthat the various changes, substitutions, and alterations could be madehereto without departing from the spirit and scope of the invention.

What is claimed is:
 1. A task circumstance processing system comprising: a memory that is configured to store task definitions that specify one or more conditions to be satisfied in a relationship between a plurality of task processes included in a task and, for each of the plurality of task processes, a recognition information that includes a pattern for recognizing execution of each of the plurality of task processes from observation data, the observation data including at least one of an image data, an audio data, and a sensor data, the image data including a captured image of the execution of the task process, the audio data including a sound associated with the execution of the task process, and the sensor data being generated by a sensor installed at an execution location; and a processor coupled to the memory, and configured to: recognize, for each of the task processes, a timing where the pattern included in the recognition information is expressed in the observation data; identify the relationship between the plurality of task processes that have been executed based on the recognized timings, determine whether or not the task that has been observed in the observation data is performed according to the task definitions by determining whether or not the identified relationship satisfies the one or more conditions to be satisfied among the plurality of task processes defined by the task definitions stored in the memory; and output the determination result.
 2. The task circumstance processing system of claim 1, wherein the recognition information is included in log information recorded by a task system employed in the execution of the plurality of task processes.
 3. The task circumstance processing system of claim 1, wherein in cases in which a relationship between new task processes has been added to the task definitions, and the recognition information of a new task process has been added to the memory, the timing is recognized where the recognition information of the new task process is expressed in the observation data, and the relationship is identified between the plurality of executed task processes that include the timing where the recognition information of the recognized new task process is expressed.
 4. The task circumstance processing system of claim 1, wherein, when recognizing the timing where the recognition information for each of the task processes is expressed, an index associated with the task process is appended to the timing in the observation data where the recognition information of each of the task processes is expressed.
 5. The task circumstance processing system of claim 4, wherein the index includes a date and a time of when the task process was executed, identification information of an observation target object, identification information of a location where observation was performed, and at least one type of observation data.
 6. The task circumstance processing system of claim 1, wherein, as the determination result, a portion is output where there is a difference between the relationship between the plurality of executed task processes and the relationship between the plurality of task processes defined by the task definitions.
 7. The task circumstance processing system of claim 6, wherein, when outputting the determination result, observation data is output corresponding to the portion having a difference.
 8. The task circumstance processing system of claim 6, wherein: when recognizing the timing where the recognition information for each of the task processes is expressed, an index for identifying the task process is appended to the timing in the observation data where the recognition information for each of the task processes is expressed; and when outputting the determination result, a listing is output of indexes appended to the observation data corresponding to any of the portions having a difference.
 9. A task circumstance processing method comprising: storing, in a memory, task definitions that specify one or more conditions to be satisfied in a relationship between a plurality of task processes included in a task and, for each of the plurality of task processes, a recognition information that includes a pattern for recognizing execution of each of the plurality of task processes from observation data, the observation data including at least one of an image data, an audio data, and a sensor data, the image data including a captured image of the execution of the task process, the audio data including a sound associated with the execution of the task process, and the sensor data being generated by a sensor installed at an execution location; recognizing, by a processor, for each of the task processes, a timing where the pattern included in the recognition information is expressed in the observation data; identifying, by the processor, a relationship between the plurality of task processes that have been executed based on the recognized timings; determining, by the processor, whether or not the task that has been observed in the observation data is performed according to the task definitions by determining whether or not the identified relationship satisfies the one or more conditions to be satisfied among the plurality of task processes defined by the task definitions stored in the memory; and outputting the determination result.
 10. The task circumstance processing method of claim 9, wherein the recognition information is included in log information recorded by a task system employed in the execution of the plurality of task processes.
 11. The task circumstance processing method of claim 9, wherein in cases in which a relationship between new task processes has been added to the task definitions, and the recognition information of a new task process has been added to the memory, the timing is recognized where the recognition information of the new task process is expressed in the observation data, and the relationship is identified between the plurality of executed task processes that include the timing where the recognition information of the recognized new task process is expressed.
 12. The task circumstance processing method of claim 9, wherein, when recognizing the timing where the recognition information for each of the task processes is expressed, an index associated with the task process is appended to the timing in the observation data where the recognition information of each of the task processes is expressed.
 13. The task circumstance processing method of claim 9, wherein, as the determination result, a portion is output where there is a difference between the relationship between the plurality of executed task processes and the relationship between the plurality of task processes defined by the task definitions.
 14. The task circumstance processing method of claim 13, wherein, when outputting the determination result, observation data is output corresponding to the portion having a difference.
 15. The task circumstance processing method of claim 13, wherein: when recognizing the timing where the recognition information for each of the task processes is expressed, an index for identifying the task process is appended to the timing in the observation data where the recognition information for each of the task processes is expressed; and when outputting the determination result, a listing is output of indexes appended to the observation data corresponding to any of the portions having a difference.
 16. A non-transitory recording medium storing a task circumstance processing program that causes a computer to execute a process, the process comprising: storing, in a memory, task definitions that specify one or more conditions to be satisfied in a relationship between a plurality of task processes included in a task and, for each of the plurality of task processes, a recognition information that includes a pattern for recognizing execution of each of the plurality of task processes from observation data, the observation data including at least one of an image data, an audio data, and a sensor data, the image data including a captured image of the execution of the task process, the audio data including a sound associated with the execution of the task process, and the sensor data being generated by a sensor installed at an execution location; recognizing, for each of the task processes, a timing where the pattern included in the recognition information is expressed in the observation data; identifying a relationship between the plurality of task processes that have been executed based on the recognized timings; determining whether or not the task that has been observed in the observation data is performed according to the task definitions by determining whether or not the identified relationship satisfies the one or more conditions to be satisfied among the plurality of task processes defined by the task definitions stored in the memory; and outputting the determination result.
 17. The non-transitory recording medium of claim 16, wherein, in the process, the recognition information is included in log information recorded by a task system employed in the execution of the plurality of task processes.
 18. The non-transitory recording medium of claim 16, wherein, in the process, in cases in which a relationship between new task processes has been added to the task definitions, and the recognition information of a new task process has been added to the memory, the timing is recognized where the recognition information of the new task process is expressed in the observation data, and the relationship is identified between the plurality of executed task processes that include the timing where the recognition information of the recognized new task process is expressed. 